Facilitating dynamic hierarchical management of queue resources in an on-demand services environment

ABSTRACT

In accordance with embodiments, there are provided mechanisms and methods for facilitating dynamic hierarchical management of queue resources in an on-demand services environment in a multi-tenant environment according to one embodiment. In one embodiment and by way of example, a method includes assigning, in runtime, by the database system, weights to at least one of a plurality of tenants and a plurality of message types. The assigned weights are capable of being dynamically scaled, in runtime, based on one or more factors. The method may further include allocating, in runtime, by the database system, resources to one or more of the plurality of tenants and one or more of the plurality of message types based on their assigned one or more weights of the weights. The allocated resources are capable of being dynamically modified, in runtime, based on scaling of the assigned weights.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

TECHNICAL FIELD

One or more implementations relate generally to data management and,more specifically, to facilitating dynamic hierarchical management ofqueue resources in an on-demand services environment.

BACKGROUND

Conventional message queueing techniques provide for a static allocationof queue resources without taking into consideration size or importanceof organizations and message types. This often leads to undesirableallocation and consumption of queue resources that do not align with therequirements (such as revenue objectives) of the business.

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem mentioned in the backgroundsection or associated with the subject matter of the background sectionshould not be assumed to have been previously recognized in the priorart. The subject matter in the background section merely representsdifferent approaches.

In conventional database systems, users access their data resources inone logical database. A user of such a conventional system typicallyretrieves data from and stores data on the system using the user's ownsystems. A user system might remotely access one of a plurality ofserver systems that might in turn access the database system. Dataretrieval from the system might include the issuance of a query from theuser system to the database system. The database system might processthe request for information received in the query and send to the usersystem information relevant to the request. The secure and efficientretrieval of accurate information and subsequent delivery of thisinformation to the user system has been and continues to be a goal ofadministrators of database systems. Unfortunately, conventional databaseapproaches are associated with various limitations.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following drawings like reference numbers are used to refer tolike elements. Although the following figures depict various examples,one or more implementations are not limited to the examples depicted inthe figures.

FIG. 1 illustrates a system having a computing device employing adynamic hierarchical queue resource allocation mechanism according toone embodiment;

FIG. 2 illustrates a dynamic hierarchical queue resource allocationmechanism 110 according to one embodiment;

FIG. 3A illustrates a transaction sequence for dynamic hierarchicalmanagement of queue resources according to one embodiment;

FIG. 3B illustrates a transaction sequence for constructing amulti-level (e.g., two-level) hierarchy for dynamic hierarchicalmanagement of queue resources according to one embodiment;

FIG. 4 illustrates a method for hierarchical allocation of queueresources for tenants and/or message types and/or a combination thereoffor dynamic hierarchical management of queue resources according to oneembodiment;

FIG. 5 illustrates a computer system according to one embodiment;

FIG. 6 illustrates an environment wherein an on-demand database servicemight be used according to one embodiment; and

FIG. 7 illustrates elements of environment of FIG. 6 and variouspossible interconnections between these elements according to oneembodiment.

SUMMARY

In accordance with embodiments, there are provided mechanisms andmethods for facilitating dynamic hierarchical management of queueresources in an on-demand services environment in a multi-tenantenvironment according to one embodiment. In one embodiment and by way ofexample, a method includes assigning, in runtime, by the databasesystem, weights to at least one of a plurality of tenants and aplurality of message types. The assigned weights are capable of beingdynamically scaled, in runtime, based on one or more factors. The methodmay further include allocating, in runtime, by the database system,resources to one or more of the plurality of tenants and one or more ofthe plurality of message types based on their assigned one or moreweights of the weights. The allocated resources are capable of beingdynamically modified, in runtime, based on scaling of the assignedweights.

While the present invention is described with reference to an embodimentin which techniques for facilitating management of data in an on-demandservices environment are implemented in a system having an applicationserver providing a front end for an on-demand database service capableof supporting multiple tenants, the present invention is not limited tomulti-tenant databases nor deployment on application servers.Embodiments may be practiced using other database architectures, i.e.,ORACLE®, DB2® by IBM and the like without departing from the scope ofthe embodiments claimed.

Any of the above embodiments may be used alone or together with oneanother in any combination. Inventions encompassed within thisspecification may also include embodiments that are only partiallymentioned or alluded to or are not mentioned or alluded to at all inthis brief summary or in the abstract. Although various embodiments ofthe invention may have been motivated by various deficiencies with theprior art, which may be discussed or alluded to in one or more places inthe specification, the embodiments of the invention do not necessarilyaddress any of these deficiencies. In other words, different embodimentsof the invention may address different deficiencies that may bediscussed in the specification. Some embodiments may only partiallyaddress some deficiencies or just one deficiency that may be discussedin the specification, and some embodiments may not address any of thesedeficiencies.

DETAILED DESCRIPTION

Methods and systems are provided for facilitating dynamic hierarchicalmanagement of queue resources in an on-demand services environment in amulti-tenant environment according to one embodiment.

Embodiments provide for facilitating dynamic hierarchical management ofqueue resources in a multi-tenant environment. It is contemplated thatmost large-scale multi-tenant queueing systems are capable of servicingtens of thousands of tenants and hundreds of millions of messages on adaily-basis and accordingly, any unfair allocation of resources, ascaused by conventional techniques, can be on a massive scale which canadversely impact implicit performance guarantees provided by the system.

Embodiments provide for a queue scheduling framework for balancingresource allocation and utilization among competing tenants, messagetypes, and a combination thereof. Embodiments provide for a hierarchicalformation of message in managing queue resources such that the queueresources are allocated and utilized based on a hierarchy of multipledimensions, such as tenants and message types. This way, resources areallocated proportionally to each tenant and each message type byconsidering any number of factors, such as size, importance, etc.,relating to the tenants and message types.

It is contemplated that embodiments and their implementations are notmerely limited to multi-tenant database system (“MTDBS”) and can be usedin other environment, such as a client-server system, a mobile device, apersonal computer (“PC”), a web services environment, etc. However, forthe sake of brevity and clarity, throughout this document, embodimentsare described with respect to a multi-tenant database system, such asSalesforce.com®, which is to be regarded as an example of an on-demandservices environment. Other on-demand services environments includeSalesforce® Exact Target Marketing Cloud™.

As used herein, a term multi-tenant database system refers to thosesystems in which various elements of hardware and software of thedatabase system may be shared by one or more customers. For example, agiven application server may simultaneously process requests for a greatnumber of customers, and a given database table may store rows for apotentially much greater number of customers. As used herein, the termquery plan refers to a set of steps used to access information in adatabase system.

Embodiments are described with reference to an embodiment in whichtechniques for facilitating management of data in an on-demand servicesenvironment are implemented in a system having an application serverproviding a front end for an on-demand database service capable ofsupporting multiple tenants, embodiments are not limited to multi-tenantdatabases nor deployment on application servers. Embodiments may bepracticed using other database architectures, i.e., ORACLE®, DB2® by IBMand the like without departing from the scope of the embodimentsclaimed.

FIG. 1 illustrates a system 100 having a computing device 120 employinga dynamic hierarchical queue resource allocation mechanism(“hierarchical mechanism”) 110 according to one embodiment. In oneembodiment, computing device 120 includes a host server computer servinga host machine for employing dynamic hierarchical queue resourceallocation mechanism (“hierarchical mechanism”) 110 for facilitatingelastic allocation of message queue resources to tenants in amulti-tiered, multi-tenant, on-demand services environment.

It is to be noted that terms like “queue message”, “job”, “query”,“request” or simply “message” may be referenced interchangeably andsimilarly, terms like “job types”, “message types”, “query type”, and“request type” may be referenced interchangeably throughout thisdocument. It is to be further noted that messages may be associated withone or more message types which may relate to or be associated with oneor more customer organizations, such as customer organizations121A-121N, where, as aforementioned, throughout this document, “customerorganizations” may be referred to as “tenants”, “customers”, or simply“organizations”. An organization, for example, may include or refer to(without limitation) a business (e.g., small business, big business,etc.), a company, a corporation, a non-profit entity, an institution(e.g., educational institution), an agency (e.g., government agency),etc.), etc., serving as a customer or client of host organization 101(also referred to as “service provider” or simply “host”) (e.g.,Salesforce®) serving as a host of hierarchical mechanism 110.

Similarly, the term “user” may refer to a system user, such as (withoutlimitation) a software/application developer, a system administrator, adatabase administrator, an information technology professional, aprogram manager, product manager, etc. The term “user” may further referto an end-user, such as (without limitation) one or more of customerorganizations 121A-N and/or their representatives (e.g., individuals orgroups working on behalf of one or more of customer organizations121A-N), such as a salesperson, a sales manager, a product manager, anaccountant, a director, an owner, a president, a system administrator, acomputer programmer, an information technology (“IT”) representative,etc.

Computing device 100 may include (without limitation) server computers(e.g., cloud server computers, etc.), desktop computers, cluster-basedcomputers, set-top boxes (e.g., Internet-based cable television set-topboxes, etc.), etc. Computing device 100 includes an operating system(“OS”) 106 serving as an interface between one or more hardware/physicalresources of computing device 100 and one or more client devices130A-130N, etc. Computing device 100 further includes processor(s) 102,memory 104, input/output (“I/O”) sources 108, such as touchscreens,touch panels, touch pads, virtual or regular keyboards, virtual orregular mice, etc.

In one embodiment, host organization 101 may further employ a productionenvironment that is communicably interfaced with client devices 130A-Nthrough host organization 101. Client devices 130A-N may include(without limitation) customer organization-based server computers,desktop computers, laptop computers, mobile computing devices, such assmartphones, tablet computers, personal digital assistants, e-readers,media Internet devices, smart televisions, television platforms,wearable devices (e.g., glasses, watches, bracelets, smartcards,jewelry, clothing items, etc.), media players, global positioningsystem-based navigation systems, cable setup boxes, etc.

In one embodiment, the illustrated multi-tenant database system 150includes database(s) 140 to store (without limitation) information,relational tables, datasets, and underlying database records havingtenant and user data therein on behalf of customer organizations 121A-N(e.g., tenants of multi-tenant database system 150 or their affiliatedusers). In alternative embodiments, a client-server computingarchitecture may be utilized in place of multi-tenant database system150, or alternatively, a computing grid, or a pool of work servers, orsome combination of hosted computing architectures may be utilized tocarry out the computational workload and processing that is expected ofhost organization 101.

The illustrated multi-tenant database system 150 is shown to include oneor more of underlying hardware, software, and logic elements 145 thatimplement, for example, database functionality and a code executionenvironment within host organization 101. In accordance with oneembodiment, multi-tenant database system 150 further implementsdatabases 140 to service database queries and other data interactionswith the databases 140. In one embodiment, hardware, software, and logicelements 145 of multi-tenant database system 130 and its other elements,such as a distributed file store, a query interface, etc., may beseparate and distinct from customer organizations (121A-121N) whichutilize the services provided by host organization 101 by communicablyinterfacing with host organization 101 via network(s) 135 (e.g., cloudnetwork, the Internet, etc.). In such a way, host organization 101 mayimplement on-demand services, on-demand database services, cloudcomputing services, etc., to subscribing customer organizations121A-121N.

In some embodiments, host organization 101 receives input and otherrequests from a plurality of customer organizations 121A-N over one ormore networks 135; for example, incoming search queries, databasequeries, application programming interface (“API”) requests,interactions with displayed graphical user interfaces and displays atclient devices 130A-N, or other inputs may be received from customerorganizations 121A-N to be processed against multi-tenant databasesystem 150 as queries via a query interface and stored at a distributedfile store, pursuant to which results are then returned to an originatoror requestor, such as a user of client devices 130A-N at any of customerorganizations 121A-N.

As aforementioned, in one embodiment, each customer organization 121A-Nis an entity selected from a group consisting of a separate and distinctremote organization, an organizational group within host organization101, a business partner of host organization 101, a customerorganization 121A-N that subscribes to cloud computing services providedby host organization 101, etc.

In one embodiment, requests are received at, or submitted to, a webserver within host organization 101. Host organization 101 may receive avariety of requests for processing by host organization 101 and itsmulti-tenant database system 150. For example, incoming requestsreceived at the web server may specify which services from hostorganization 101 are to be provided, such as query requests, searchrequest, status requests, database transactions, graphical userinterface requests and interactions, processing requests to retrieve,update, or store data on behalf of one of customer organizations 121A-N,code execution requests, and so forth. Further, the web-server at hostorganization 101 may be responsible for receiving requests from variouscustomer organizations 121A-N via network(s) 135 on behalf of the queryinterface and for providing a web-based interface or other graphicaldisplays to one or more end-user client devices 130A-N or machinesoriginating such data requests.

Further, host organization 101 may implement a request interface via theweb server or as a stand-alone interface to receive requests packets orother requests from the client devices 130A-N. The request interface mayfurther support the return of response packets or other replies andresponses in an outgoing direction from host organization 101 to one ormore client devices 130A-N.

It is to be noted that any references to software codes, data and/ormetadata (e.g., Customer Relationship Model (“CRM”) data and/ormetadata, etc.), tables (e.g., custom object table, unified indextables, description tables, etc.), computing devices (e.g., servercomputers, desktop computers, mobile computers, such as tabletcomputers, smartphones, etc.), software development languages,applications, and/or development tools or kits (e.g., Force.com®,Force.com Apex™ code, JavaScript™, jQuery™, Developerforce™,Visualforce™, Service Cloud Console Integration Toolkit™ (“IntegrationToolkit” or “Toolkit”), Platform on a Service™ (“PaaS”), Chatter®Groups, Sprint Planner®, MS Project®, etc.), domains (e.g., Google®,Facebook®, LinkedIn®, Skype®, etc.), etc., discussed in this documentare merely used as examples for brevity, clarity, and ease ofunderstanding and that embodiments are not limited to any particularnumber or type of data, metadata, tables, computing devices, techniques,programming languages, software applications, software developmenttools/kits, etc.

It is to be noted that terms like “node”, “computing node”, “server”,“server device”, “cloud computer”, “cloud server”, “cloud servercomputer”, “machine”, “host machine”, “device”, “computing device”,“computer”, “computing system”, “multi-tenant on-demand data system”,and the like, may be used interchangeably throughout this document. Itis to be further noted that terms like “code”, “software code”,“application”, “software application”, “program”, “software program”,“package”, “software code”, “code”, and “software package” may be usedinterchangeably throughout this document. Moreover, terms like “job”,“input”, “request”, and “message” may be used interchangeably throughoutthis document.

FIG. 2 illustrates a dynamic hierarchical queue resource allocationmechanism 110 according to one embodiment. In one embodiment,hierarchical mechanism 110 may include a number of components, such as(without limitation and not necessarily in this order): administrationengine 201 and hierarchical allocation engine (“allocation engine”) 211.In one embodiment, administration engine may include a number ofcomponents, such as (without limitation and not necessarily in thisorder) request/query logic 203, authentication logic 205, andcommunication/compatibility logic 207. Similarly, in one embodiment,allocation engine 211 may include a number of components, such as(without limitation and not necessarily in this order):detecting/monitoring logic (“detecting logic”) 213 including fair usagemonitoring module (“fair usage module”) 215; evaluation/selection logic217; aggregating and fetching logic 219; weight assignment logic(“weight logic”) 221; and resource allocation logic (“resource logic”)223.

Throughout this document, terms like “framework”, “mechanism”, “engine”,“logic”, “component”, “module”, “tool”, and “builder” may be referencedinterchangeably and include, by way of example, software, hardware,and/or any combination of software and hardware, such as firmware.Further, any use of a particular brand, word, or term, such as“hierarchical” or “hierarchy”, “fair usage”, “two-level” or“multi-level”, “weight” or “adjusted weight”, “queue”, “allocation”,“registry”, etc., and/or any of their forms, such as “queuing”,“allocating”, “weighting”, “registering”, etc., should not be read tolimit embodiments to software or devices that carry that label inproducts or in literature external to this document.

As aforementioned, with respect to FIG. 1, any number and type ofrequests and/or queries may be received at or submitted to request/querylogic 203 for processing. For example, incoming requests may specifywhich services from computing device 120 are to be provided, such asquery requests, search request, status requests, database transactions,graphical user interface requests and interactions, processing requeststo retrieve, update, or store data, etc., on behalf of one or moreclient devices 130A-N, code execution requests, and so forth.

In one embodiment, computing device 120 may implement request/querylogic 203 to serve as a request/query interface via a web server or as astand-alone interface to receive requests packets or other requests fromthe client devices 130A-N. The request interface may further support thereturn of response packets or other replies and responses in an outgoingdirection from computing device 120 to one or more client devices130A-N.

Similarly, request/query logic 203 may serve as a query interface toprovide additional functionalities to pass queries from, for example, aweb service into the multi-tenant database system for execution againstdatabase(s) 140 and retrieval of customer data and stored recordswithout the involvement of the multi-tenant database system or forprocessing search queries via the multi-tenant database system, as wellas for the retrieval and processing of data maintained by otheravailable data stores of the host organization's production environment.Further, authentication logic 205 may operate on behalf of the hostorganization, via computing device 120, to verify, authenticate, andauthorize, user credentials associated with users attempting to gainaccess to the host organization via one or more client devices 130A-N.

In one embodiment, computing device 120 may include a server computerwhich may be further in communication with one or more databases orstorage repositories, such as database(s) 140, which may be locatedlocally or remotely over one or more networks, such as network(s) 235(e.g., cloud network, Internet, proximity network, intranet, Internet ofThings (“IoT”), Cloud of Things (“CoT”), etc.). Computing device 100 isfurther shown to be in communication with any number and type of othercomputing devices, such as client computing devices 130A-N, over one ormore networks, such as network(s) 140.

As aforementioned, embodiments provide for a hierarchical structure viahierarchical mechanism 110 to facilitate a multi-level (e.g., two-level)hierarchy in a multi-tenant environment. It is contemplated thatembodiments are not limited to any number of levels of hierarchy;however, for the sake of brevity, clarity, and ease of understanding, ahierarchy based on two-levels is discussed throughout this document. Forexample, at a top level of the two-level hierarchy, an amount ofresources is determined and allocated to each message type. Within eachmessage type, the allocated resources are further divided amongcompeting tenants. However, as aforementioned, this two-levelhierarchical framework may be extended into multiple levels if, forexample, a tenant may want to further distinguish its messages intogeographical regions, business functions, etc. In one embodiment, anamount of resources (e.g., application thread time) is determined to bedistributed to each competing message type, tenant, or a combinationthereof, by considering factors, such as size, importance, etc.

It is contemplated that in many multi-tenant environments, there is afixed number of application server or thread resources that may beavailable to be allocated and thus, in one embodiment, using resourcelogic 223, this fixed pool of application servers or thread resourcesmay be allocated across two levels, such as first, to competing messagetypes and second, to competing tenant within or associated with aparticular message type. In one embodiment, resource allocation may beperformed in a hierarchical manner in that thread resources are firstdistributed at a coarse grained level among different message types andonce each message type has received a distribution of resources, thoseresources may be further divided amount competing tenants from the samemessage type as facilitated by resource logic 223 of hierarchicalallocation engine 211. As aforementioned, using hierarchical allocationengine 211, a fair division of resources is performed across competingtenants within a message type, while various resource assignments aretaken into consideration based on any number of factors, such asbusiness policy decisions that favor one message type or one tenant'smessage over other tenants' messages, etc.

For brevity, clarity, and ease of understanding, for example, supposethere is a fixed number of application servers with X hours of thread(compute) capacity to be offered on a daily-basis. Now, consider that acompany has multiple departments (that are represented by differentclouds) with engineers who wish to have a fair share of the X hours ofthe available thread capacity every day. In this case, assuming thatsales cloud, service cloud, and reliability cloud all want a piece ofthis capacity and further assuming, each team is equally important, eachcloud is allocated an X/3 hours of thread capacity. Let us assume,within each cloud, there are various teams; for example, the reliabilitycloud may include a core data services team, message queue team, andservice protection team. Thus, thread capacity would have to be furtherdivided with the reliability cloud to accommodate the needs of each teamwithin that cloud. Assuming each team within the reliability cloud isequally important, each team may be allocated a share of X/9 hours ofthread time.

This computation may get a bit complicated if, for example, one team,such as message queue team, is identified by the company as having ahigher priority over other teams and thus it needs to be allocated morethread capacity than other teams (such as twice as much as other teams)under the reliability cloud. In this case, message queue team mayreceive X/6 (X/3*½) hours of thread capacity while the other two teamsmay receive X/12 hours each. To complete this analogy, for example,hierarchical mechanism 110 may allow for a two-level hierarchicalallocation with departments or clouds denoting message types and teamswithin each cloud denoting tenants. It is contemplated that in practice,resources may be allocated at a finer time scale (e.g., minutes) and atthe granularity of the number of application servers.

In one embodiment, detecting/monitoring logic 213 may be used tocontinuously, and in runtime, monitor the usage performance of anynumber and type of tenants and/or message types so that anyover-consumption and/or under-consumption of the resources by one ormore of the tenants and/or message types may be detected. In oneembodiment, fair usage module 215 of detecting/monitoring logic 213 maybe used to more specifically monitor those tenants and/or message typesthat may be nearing over-consumption or under-consumption of theirallocated resources to ensure that fair usage is continuouslymaintained.

In one embodiment, the hierarchical framework provided by hierarchicalmechanism 110 may offer a resource allocation scheme that is based on,for example, a weight scale to assign weights to each message type andeach tenant based on any number of factors, such as overall importance,priority over their counterparts, etc., as facilitated by weight logic221. For example and in one embodiment, weight logic 221 may be used tojudge each tenant and/or message type based on a weight scaling factorwhere the absolute value of the weight is not of matter and rather, itis the relative difference between the weights when compared with eachother is of matter. For example, a first tenant that is assigned aweight of 2 and a second tenant is assigned a weight of 1 by weightlogic 221, the first tenant (e.g., Coke®) may be regarded as twice asimportant as the first tenant and thus, for example, may be allocated66% of the resources while the second tenant (e.g., Pepsi®) may beallocated 33% of the resources by resource allocation logic 223 and asrecommended by fair usage module 215 to ensure fair usage of resources.

It is to be noted that Coke® received twice the share of Pepsi®'s threadresources. Now let us suppose there is a third tenant (e.g., Fanta®)submits its messages and Fanta® has a weight of 3. In this case, Fanta®may be recommended by fair usage module 215 to receive an appropriateamount of resources and accordingly, Fanta® is allocated, via resourceallocation logic 223, three times as Pepsi® and 50% more than Coke®,such as Pepsi® at 16.7%, Coke® at 33.3%, and Fanta® at 50%. It is to benoted that in one embodiment, the amount of thread resources allocatedto each tenant changes as the function of the number of competitortenants and the relative weight of each competitor tenant changes overtime.

It is contemplated that various message types may be associated with thetenants, such as DASHBOARD and APEX_FUTURE, where the three tenants maybe weighted as follows: Pepsi® at 1, Coke® at 2, and Fanta® at 3.Considering, for example, Pepsi® has DASHBOARD messages while Pepsi®,Coke®, and Fanta® all have APEX_FUTURE messages that are pendingprocessing. For brevity and simplicity, assuming that all message typesare created equal, such as in the aggregate, DASHBOARD messages receivethe same thread time as APEX_FUTURE messages. In uniform resourceallocation, the hierarchy is not considered and each message type andtenant combination is assigned the same fraction of resources, such asPepsi®, DASHBOARD at 25% of thread resources and similarly, Pepsi®,APEX_FUTURE at 25% of thread resources, Coke®, APEX_FUTURE at 25% ofthread resources, and Fanta®, APEX_FUTURE at 25% of thread resources.

In one embodiment, hierarchical mechanism 110 offers a hierarchicalallocation of resources even when message types come into considerationalong with tenants to continue to ensure fair allocation and usage ofresources by message types and/or tenants. For example, when distributedover a two-level hierarchy, such as first across competing message typesand then across competing tenants within the message type, thedistribution of resources may appear as follows: at top, such as messagetypes, DASHBOARD may get 50% of thread resources, and APEX_FUTURE mayalso get 50% of thread resources. Since both message types carry thesame importance as evaluated and selected by evaluation/selection logic217, both message types are assigned the same weight, such as 1:1, byweight logic 221 and equally allocated their resources, such as 50%-50%,by resource logic 223.

Next, for example and in one embodiment, these resources assigned to themessage types may be further divided among competing tenants withinAPEX_FUTURE (since this message type is shared by all three tenants inour example) according to the three tenants' correspondingly assignedweights as determined by evaluation/selection logic 217 and assigned byweight logic 221, as previously mentioned, such as Pepsi® at 16.7%,Coke® at 33.3%, and Fanta® at 50%. In one embodiment, using thisbreakdown of weights assigned to the three tenants, resource logic 223may then allocate the tenants their corresponding thread resources foreach message type, such as Pepsi®, DASHBOARD at 50%, Pepsi®, APEX_FUTUREat 8.35%, Coke®, APEX_FUTURE at 16.65%, and Fanta®, APEX_FUTURE at 25%.Stated differently, Pepsi® receives its entire share of resourcesallocated to DASHBOARD which, in this example, is 50% since only Pepsi®has DASHBOARD. However, given that APEX_FUTURE is shared by all threetenants in this example, Pepsi®, APEX_FUTURE receives only 16.7% of 50%of the resources, which translates into 8.35% of the overall threadresources. Similarly, Coke®, APEX_FUTURE receives 16.65% and Fanta®,APEX_FUTURE receives 25% of the total global thread resources.

To continue with the example, suppose Pepsi® no longer has APEX_FUTUREtype of messages to process, this change may be detected bydetecting/monitoring logic 213, monitored by fair usage module 215 toensure that fair allocation of thread resources is ensured, evaluated byevaluation/selection logic 217 and accordingly, any weight allocationmay be changed (as necessary) by weight logic 221 and the allocation ofthread resources may be adjusted and reallocated by resource logic 223.In one embodiment, as illustrated with respect to resource utilizationaggregator 303 of FIG. 3A, aggregating and fetching logic 219 may beused to fetch queuing time for each message type and tenant and mayshare the collected information with one or more components ofhierarchical allocation engine 211, such as evaluation/selection logic217 for further evaluation, weight logic 221 for weight assignment, fairusage module 215 to determine continuing fair usage of resources, etc.Referring back to Pepsi® no longer having APEX_FUTURE type messages toprocess, the composition may change to Pepsi®, DASHBOARD at 50%, Coke®,APEX_FUTURE at 20%, and Fanta®, APEX_FUTURE at 30% as facilitated byresource logic 223.

It is to be noted that in one embodiment, the resources freed by Pepsiare then redistributed among tenants under APEX_FUTURE to ensure thatAPEX_FUTURE messages continue to receive the same fraction of totalresources as DASHBOARD messages. Similarly, for example, if DASHBOARD nolonger has any messages, then the entire share of thread resources maybe redistributed to APEX_FUTURE (e.g., resource logic 223 is workconserving).

It is contemplated that often, whether a tenant has messages to processis not a binary decision, such as Pepsi® may have fewer APEX_FUTUREmessages than the available thread capacity for a given time period. Inthis case, Pepsi®'s weight may be adjusted accordingly to match theamount of thread resources that it may consume maximally. For example,assuming a factor of 1.0 denotes that a tenant has enough work toconsume available thread resources in its entirety for the next timeperiod (also referred to as “work consumption factor”), let Pepsi® haveenough work to consume 50% of available capacity, then the workconsumption for the three tenants with respect to APEX_FUTURE may be asfollows: Pepsi® at 0.5, Coke® at 1.0, and Fanta® at 1.0.

Similarly, the weight assigned to each tenant may be adjusted by weightlogic 221 which essentially translates into a reduction in Pepsi®'sweight based on its actual resource needs as opposed to the maximalshare of the capacity, such as Pepsi® at 1*0.5=0.5, Coke® at 2*1.0=2,and Fanta® at 3*1.0=3. Given this adjusted weight, the redistributionportion of Pepsi®'s thread resources with respect to Coke® and Fanta®may appear as follows: Pepsi®, DASHBOARD at 50% of total threadresources, Pepsi®, APEX_FUTURE at 4.5% of total thread resources, Coke®,APEX_FUTURE at 18.2% of total thread resources, and Fanta®, APEX_FUTUREat 27.3% of total thread resources.

In one embodiment, evaluation/selection logic 217 may be used to performvarious evaluation and computation tasks to accurately lead todetermination of weight and how they may be seeded along with whether atenant and/or a message type has jobs to process, etc. For example, uponperforming various computations, weight logic 221 may offer an interface(e.g., QueueWeightScaling) to assign and seed the weight and determinethe actual fraction of resources that is to be allocated based on theassigned weight to each tenant and/or message type. Similarly, uponknowing the weight and the corresponding fraction of resources that isto be allocated, resource logic 223 may then allocate the fraction tothe tenant and/or message type according to their assigned weight. It iscontemplated that any number and type of interface may be provided andthat embodiments are not limited to any particular interface or form ofinterface, such as UniformQueueWeightScaling may be used for providinguniform allocation of resources, whileHierarchicalOrgAwareQueueWeightScaling may be used for providinghierarchical allocation of resources according to one embodiment.

In one embodiment, the initial weights associated with tenants and/ormessage types may be the same; however, fair usage module 215 maymonitor various, often dynamically changing, business policies or policyrules associated with the tenants and/or message types so that theresources may be fairly allocated. Accordingly, for example and in oneembodiment, based on such business policies, the initial weights may bealtered as computed by evaluation/selection logic 217 and seeded orassigned by weight logic 221. In one embodiment, these business policiesmay be stored in one or more configuration files at database(s) 140.Further, for example and in one embodiment, shared queues for messagetypes may also be supported and as such, the highest weighted messagetype may be used to determine the weight for the shared queues.

Similarly, in one embodiment, those tenants that are not detected oridentified offending/misbehaving tenants or victim/starving tenants bydetecting/monitoring logic 213, such tenants are regarded as havinglower traffic (e.g., needing fewer threads) and may share the same queuewithin a message type. For purposes of resource allocation, these sharedtenant queues are regarded as a single logical tenant with a singleassigned weight, such as 1. For example with regard to DASHBOARD messagetype, Pepsi®, DASHBOARD may be assigned weight 1, Coke®, DASHBOARD maybe assigned weight 2, and Shared Tenants, DASHBOARD may be assignedweight 1 (being treated as a single logical tenant) and accordingly,resource distribution may be as follows: Pepsi®, DASHBOARD at 25% ofresources, Coke®, DASHBOARD at 50% of resources, and Shared Tenants,DASHBOARD at 25% of resources.

In some embodiments, to determine whether a tenant, a message type, or acombination thereof has any messages to process (such as to determinewhether any resources are to be redistributed among other tenants),reliance is placed on queuing time measurements (e.g., time spent bymessages waiting on a queue, etc.) which are collected from metering ona per-tenant or a per-message type basis as facilitated by aggregatingand fetching logic 219. For example, using the previous example, ifPepsi® has a queuing time equally zero, it may then be considered ashaving no messages and as such, its resources may be redistributed asfollows: Coke®, DASHBOARD at 66% of resources, and Shared Tenants,DASHBOARD at 33% of resources.

It is contemplated that in an actual or real-life multi-tenantenvironment, hierarchical mechanism 110 may regard queuing times in amore nuanced manner (e.g., it may not be just an all or nothingdecision); for example, queueing time may be used to approximate workconsumption factor described above to adjust the weight of a tenant.Thus, for example, if the queueing time for Pepsi® is less than that ofCoke®, then Pepsi®'s weight may be adjusted downwards accordingly toallow for redistribution of thread resources to other tenants, asnecessitated. In one embodiment, queuing time may be used as anapproximation of the amount of pending messages (e.g., work) fromPepsi®, where the approximation may be used if querying a precise numberof messages on the queue may be expensive or inefficient to perform onthe system and further, the queuing time may be a reliable predictor ofpending messages.

Further, in one embodiment, database(s) 140 may include any number andtype of tables and cache, such as routing table 301, tenant and messagetype registry 311, and cache 305 including two-level hierarchy cache 307and adjusted weights cache 309, etc., as illustrated and will be furtherdiscussed with respect to FIG. 3A. In one embodiment, discovery ofhierarchy may be performed via detecting and monitoring logic 213 whichmay loop through each queue and/or each message type and tenantcombination which may then be used by evaluation/selection logic 217 todetermine weight scaling factors.

Further, aggregating and fetching logic 219 may be used to group thetenants relating to the same message type together under the samehierarchy for, for example, indexing and efficient look up using one ormore of the aforementioned tables and cache at database(s) 140. Forexample and in one embodiment, caching of message type hierarchy may beperformed to maintain a cache of the sum of weights of, for example, allthose tenants having a non-zero queuing time within the same messagetype. This sum may be used to determine a number of competing tenantswith messages to process and their corresponding weights. This cachingmay refer to and used as an optimization to avoid computing the sum whenquerying the resource allocation for each message type, tenant, and/or acombination thereof.

In one embodiment, evaluation/selection logic 217 may be used forresource fraction computation via an interface, such as allowing forquerying the fraction of resources allocated to each message type,tenant, and/or a combination thereof. For example and in one embodiment,to compute, a fraction of resource assigned to each message type inquestion is determined, where this fraction is denoted by M (e.g., viaan interface, such as getScalingFactorForMessageType, etc.). Next, theweight of the tenant in question may be fetched via aggregating andfetching logic 219 (e.g., via an interface, such asgetScalingFactorForOrg, etc.), where the weight may be referred to as Oand is adjusted by queuing time. If, for example, the tenant hasqueueing time, T, then the adjusted weight may be computed byevaluation/selection logic 217 as O*T. Then, a sum of the adjustedweight, S, of all those tenants with messages to process or non-zeroqueuing time may be obtained and grouped under the message type inquestion. The fraction of the resources assigned to a specific messagetype, tenant, or a combination thereof, may then be computed as M*O*T/S.

Communication/compatibility logic 207 may facilitate the ability todynamically communicate and stay configured with any number and type ofsoftware/application developing tools, models, data processing servers,database platforms and architectures, programming languages and theircorresponding platforms, etc., while ensuring compatibility withchanging technologies, parameters, protocols, standards, etc.

It is contemplated that any number and type of components may be addedto and/or removed from hierarchical mechanism 110 to facilitate variousembodiments including adding, removing, and/or enhancing certainfeatures. It is contemplated that embodiments are not limited to anyparticular technology, topology, system, architecture, and/or standardand are dynamic enough to adopt and adapt to any future changes.

FIG. 3A illustrates a transaction sequence 300 for dynamic hierarchicalmanagement of queue resources according to one embodiment. Transactionsequence 300 may be performed by processing logic that may comprisehardware (e.g., circuitry, dedicated logic, programmable logic, etc.),software (such as instructions run on a processing device), or acombination thereof. In one embodiment, transaction sequence 300 may beperformed or facilitated by hierarchical mechanism 110 of FIGS. 1-2. Theprocesses of transaction sequence 300 are illustrated in linearsequences for brevity and clarity in presentation; however, it iscontemplated that any number of them can be performed in parallel,asynchronously, or in different orders. Further, for brevity, clarity,and ease of understanding, many of the components and processesdescribed with respect to FIGS. 1-2 may not be repeated or discussedhereafter.

In one embodiment, transaction sequence 300 is shown to employ one ormore tables and/or cache components which may part of database(s) 140 ofFIG. 2. For example and in one embodiment, tables and/or cachecomponents may include (without limitation) routing table 301, tenantand message type registry 311, and cache 305 including two-levelhierarchy cache 307 and adjusted weights cache 309. In one embodiment,as illustrated, routing table 301 may hold any amount and type of datarelating to queues, tenants, message types, etc., and may be used toprovide, from within routing table 301, any snapshots of queues for eachmessage type and/or tenant and/or a combination thereof to resourceallocator 313 being facilitated by resource allocation logic 223 of FIG.2.

Similarly, in one embodiment, tenant and message type registry 311 mayhold any amount and type of data relating to queues, tenants, messagetypes, etc., such as business policies (and/or business policy rules)and/or any credentials relating to each queue, tenant, and message type,etc., may be registered for identification of their priorities,importance, etc., and fetching and/or other purposes. In one embodiment,tenant and message type registry 311 may be used to fetch or collectpriorities relating to message types and/or tenants and/or a combinationthereof and forward that information on to resource allocator 313. Inone embodiment, resource utilization aggregator 303, as facilitated byaggregating and fetching logic 219, may be used for fetch queuing timefor each message type and/or tenant and/or a combination thereof andforwarding that information on to resource locator 313.

At resource locator 313, a determination may be made as to whetherthread resources are to be allocated based on a uniform manner or in ahierarchical manner. As aforementioned, in some embodiments, threadresources are allocated in a uniform manner using uniform allocator 315of resource allocator 313. However, as described throughout thisdocument, in some embodiments, thread resources may be allocated basedon a multi-level hierarchical manner (such as based on a two-levelhierarchy, as discussed throughout this document) via hierarchyallocator 317 as facilitated by various components of hierarchicalallocation engine 211.

As illustrated, resource allocator 313 may be further in communicationwith cache 315 in communicating any amount and type of information, suchas resource allocator 313 may provide updated list of message typesand/or tenants to cache 315 to be stored at two-level hierarchy cache307. It is contemplated and as previously noted, embodiments are notlimited to merely two-levels of hierarchy and thus, two-level hierarchycache 307 is provided as an example for consistency, brevity, andclarity, and that it may be a multi-level hierarchy having any numberand type of levels to support a multi-level hierarchical allocation ofresources. Further, in one embodiment, resource allocator 313 may befurther in communication with cache 315 to receive weights and any datarelating to the weights associated with tenants and/or message typesand/or a combination thereof, as facilitated by weight assignment logic221, as fetched from adjusted weights cache 309 and provided on toresource allocator 313. In one embodiment, resource allocator 313 isfurther in communication with fair usage monitor 317, as facilitated byfair usage monitoring module 215, to enforce tier promotion and/ordemotion for queues and to ensure and maintain a fair allocation and useof thread resources between tenants, message types, and a combinationthereof, throughout the multi-tenant system.

FIG. 3B illustrates a transaction sequence 350 for constructing amulti-level (e.g., two-level) hierarchy for dynamic hierarchicalmanagement of queue resources according to one embodiment. Transactionsequence 350 may be performed by processing logic that may comprisehardware (e.g., circuitry, dedicated logic, programmable logic, etc.),software (such as instructions run on a processing device), or acombination thereof. In one embodiment, transaction sequence 350 may beperformed or facilitated by hierarchical mechanism 110 of FIGS. 1-2. Theprocesses of transaction sequence 350 are illustrated in linearsequences for brevity and clarity in presentation; however, it iscontemplated that any number of them can be performed in parallel,asynchronously, or in different orders. Further, for brevity, clarity,and ease of understanding, many of the components and processesdescribed with respect to FIGS. 1-2 may not be repeated or discussedhereafter.

As illustrated, in one embodiment, transaction sequence 350 includes anumber of components, such as routing table 301, resource allocator 313,and two-level hierarchy cache 307, in communication with each other.Transaction sequence 350 begins at routing table 301 with collecting alist of message type level routing rules 351 is collected andcommunicated with resource allocator 313, where a node is created thatsub-divides the message type into a tenant-level hierarchy 353 andcommunicated on to two-level hierarchy cache 307. Similarly, any weightscaling factor for the message type and the node is communicated to andstored 355 at two-level hierarchy cache 307. Further, in one embodiment,at resource allocator 313, the aforementioned processes may be repeatedfor each message type level rule 357.

Similarly, as with message types, in one embodiment, a list of tenantlevel routing rules is collected 359 and communicated on to resourceallocator 313 which fetches the tenant level hierarchy for thecorresponding message type 361 to two-level hierarchy cache 307. Atresource allocator 313, a tenant is added to the hierarchy along with aweight scaling factor for the tenant 363. Any tenant level hierarchy forthe message type is updated 367 and communicated on to two-levelhierarchy cache 307 and, in some embodiments, the aforementionedprocesses may be repeated for each tenant level routing rule at resourceallocator 313.

FIG. 4 illustrates a method 400 for hierarchical allocation of queueresources for tenants and/or message types and/or a combination thereoffor dynamic hierarchical management of queue resources according to oneembodiment. Method 400 may be performed by processing logic that maycomprise hardware (e.g., circuitry, dedicated logic, programmable logic,etc.), software (such as instructions run on a processing device), or acombination thereof. In one embodiment, method 400 may be performed orfacilitated by hierarchical mechanism 110 of FIGS. 1-2. The processes ofmethod 400 are illustrated in linear sequences for brevity and clarityin presentation; however, it is contemplated that any number of them canbe performed in parallel, asynchronously, or in different orders.Further, for brevity, clarity, and ease of understanding, many of thecomponents and processes described with respect to FIGS. 1-2 may not berepeated or discussed hereafter.

Method 400 may begin at block 401 with fetching queuing time for one ormore tenants, message types, and/or a combination thereof. At block 403,a determination is made as to whether any of the tenants and/or messagetypes in queues has zero queuing time and no messages to process. Ifyes, at block 405, such tenants and/or message types are allocated zeroresources since they do not appear to need any resources as evident fromtheir zero queuing time. If, however, there is a tenant and/or messagetype that is detected to be involved in message processing needingresources, any available and relevant business policies and/or weightassignments may be accessed to determine and fetch importance levelsand/or scaling factors associated with the detected message type (J) atblock 407. Similarly, at block 409, in one embodiment, the importancelevel and/or scaling factor for the detected tenant (T) is fetched.

At block 411, a determination is made as to whether the adjusted weightsum (S) is cached. If not, a sum (Q*T) for combinations (of tenants)with the same message type is performed at block 413. The process maythen continue at block 415 with writing S for the message type to theadjusted weights cache, such as adjusted weights cache 309 of FIGS.3A-3B. Upon writing S, as in block 415, or, referring to block 411, ifthe adjusted weight sum (S) is cached, S is fetched from adjustedweights cache at block 417. At block 419, a fraction (J*T*Q/S) of theresources is allocated to the tenant and/or message type and/or acombination thereof at block 419.

FIG. 5 illustrates a diagrammatic representation of a machine 500 in theexemplary form of a computer system, in accordance with one embodiment,within which a set of instructions, for causing the machine 500 toperform any one or more of the methodologies discussed herein, may beexecuted. Machine 500 is the same as or similar to computing devices120, 130A-N of FIG. 1. In alternative embodiments, the machine may beconnected (e.g., networked) to other machines in a network (such as hostmachine 120 connected with client machines 130A-N over network 135 ofFIG. 1), such as a cloud-based network, Internet of Things (IoT) orCloud of Things (CoT), a Local Area Network (LAN), a Wide Area Network(WAN), a Metropolitan Area Network (MAN), a Personal Area Network (PAN),an intranet, an extranet, or the Internet. The machine may operate inthe capacity of a server or a client machine in a client-server networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment or as a server or series of servers within anon-demand service environment, including an on-demand environmentproviding multi-tenant database storage services. Certain embodiments ofthe machine may be in the form of a personal computer (PC), a tablet PC,a set-top box (STB), a Personal Digital Assistant (PDA), a cellulartelephone, a web appliance, a server, a network router, switch orbridge, computing system, or any machine capable of executing a set ofinstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine is illustrated,the term “machine” shall also be taken to include any collection ofmachines (e.g., computers) that individually or jointly execute a set(or multiple sets) of instructions to perform any one or more of themethodologies discussed herein.

The exemplary computer system 500 includes a processor 502, a mainmemory 504 (e.g., read-only memory (ROM), flash memory, dynamic randomaccess memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM(RDRAM), etc., static memory such as flash memory, static random accessmemory (SRAM), volatile but high-data rate RAM, etc.), and a secondarymemory 518 (e.g., a persistent storage device including hard disk drivesand persistent multi-tenant data base implementations), whichcommunicate with each other via a bus 530. Main memory 504 includesemitted execution data 524 (e.g., data emitted by a logging framework)and one or more trace preferences 523 which operate in conjunction withprocessing logic 526 and processor 502 to perform the methodologiesdiscussed herein.

Processor 502 represents one or more general-purpose processing devicessuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processor 502 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processor 502 may alsobe one or more special-purpose processing devices such as an applicationspecific integrated circuit (ASIC), a field programmable gate array(FPGA), a digital signal processor (DSP), network processor, or thelike. Processor 502 is configured to execute the processing logic 526for performing the operations and functionality of hierarchicalmechanism 110 as described with reference to FIG. 1 other figuresdiscussed herein.

The computer system 500 may further include a network interface card508. The computer system 500 also may include a user interface 510 (suchas a video display unit, a liquid crystal display (LCD), or a cathoderay tube (CRT)), an alphanumeric input device 512 (e.g., a keyboard), acursor control device 514 (e.g., a mouse), and a signal generationdevice 516 (e.g., an integrated speaker). The computer system 500 mayfurther include peripheral device 536 (e.g., wireless or wiredcommunication devices, memory devices, storage devices, audio processingdevices, video processing devices, etc. The computer system 500 mayfurther include a Hardware based API logging framework 534 capable ofexecuting incoming requests for services and emitting execution dataresponsive to the fulfillment of such incoming requests.

The secondary memory 518 may include a machine-readable storage medium(or more specifically a machine-accessible storage medium) 531 on whichis stored one or more sets of instructions (e.g., software 522)embodying any one or more of the methodologies or functions ofhierarchical mechanism 110 as described with reference to FIG. 1,respectively, and other figures discussed herein. The software 522 mayalso reside, completely or at least partially, within the main memory504 and/or within the processor 502 during execution thereof by thecomputer system 500, the main memory 504 and the processor 502 alsoconstituting machine-readable storage media. The software 522 mayfurther be transmitted or received over a network 520 via the networkinterface card 508. The machine-readable storage medium 531 may includetransitory or non-transitory machine-readable storage media.

Portions of various embodiments may be provided as a computer programproduct, which may include a computer-readable medium having storedthereon computer program instructions, which may be used to program acomputer (or other electronic devices) to perform a process according tothe embodiments. The machine-readable medium may include, but is notlimited to, floppy diskettes, optical disks, compact disk read-onlymemory (CD-ROM), and magneto-optical disks, ROM, RAM, erasableprogrammable read-only memory (EPROM), electrically EPROM (EEPROM),magnet or optical cards, flash memory, or other type ofmedia/machine-readable medium suitable for storing electronicinstructions.

The techniques shown in the figures can be implemented using code anddata stored and executed on one or more electronic devices (e.g., an endstation, a network element). Such electronic devices store andcommunicate (internally and/or with other electronic devices over anetwork) code and data using computer-readable media, such asnon-transitory computer-readable storage media (e.g., magnetic disks;optical disks; random access memory; read only memory; flash memorydevices; phase-change memory) and transitory computer-readabletransmission media (e.g., electrical, optical, acoustical or other formof propagated signals—such as carrier waves, infrared signals, digitalsignals). In addition, such electronic devices typically include a setof one or more processors coupled to one or more other components, suchas one or more storage devices (non-transitory machine-readable storagemedia), user input/output devices (e.g., a keyboard, a touchscreen,and/or a display), and network connections. The coupling of the set ofprocessors and other components is typically through one or more bussesand bridges (also termed as bus controllers). Thus, the storage deviceof a given electronic device typically stores code and/or data forexecution on the set of one or more processors of that electronicdevice. Of course, one or more parts of an embodiment may be implementedusing different combinations of software, firmware, and/or hardware.

FIG. 6 illustrates a block diagram of an environment 610 wherein anon-demand database service might be used. Environment 610 may includeuser systems 612, network 614, system 616, processor system 617,application platform 618, network interface 620, tenant data storage622, system data storage 624, program code 626, and process space 628.In other embodiments, environment 610 may not have all of the componentslisted and/or may have other elements instead of, or in addition to,those listed above.

Environment 610 is an environment in which an on-demand database serviceexists. User system 612 may be any machine or system that is used by auser to access a database user system. For example, any of user systems612 can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Asillustrated in herein FIG. 6 (and in more detail in FIG. 7) user systems612 might interact via a network 614 with an on-demand database service,which is system 616.

An on-demand database service, such as system 616, is a database systemthat is made available to outside users that do not need to necessarilybe concerned with building and/or maintaining the database system, butinstead may be available for their use when the users need the databasesystem (e.g., on the demand of the users). Some on-demand databaseservices may store information from one or more tenants stored intotables of a common database image to form a multi-tenant database system(MTS). Accordingly, “on-demand database service 616” and “system 616”will be used interchangeably herein. A database image may include one ormore database objects. A relational database management system (RDMS) orthe equivalent may execute storage and retrieval of information againstthe database object(s). Application platform 618 may be a framework thatallows the applications of system 616 to run, such as the hardwareand/or software, e.g., the operating system. In an embodiment, on-demanddatabase service 616 may include an application platform 618 thatenables creation, managing and executing one or more applicationsdeveloped by the provider of the on-demand database service, usersaccessing the on-demand database service via user systems 612, or thirdparty application developers accessing the on-demand database servicevia user systems 612.

The users of user systems 612 may differ in their respective capacities,and the capacity of a particular user system 612 might be entirelydetermined by permissions (permission levels) for the current user. Forexample, where a salesperson is using a particular user system 612 tointeract with system 616, that user system has the capacities allottedto that salesperson. However, while an administrator is using that usersystem to interact with system 616, that user system has the capacitiesallotted to that administrator. In systems with a hierarchical rolemodel, users at one permission level may have access to applications,data, and database information accessible by a lower permission leveluser, but may not have access to certain applications, databaseinformation, and data accessible by a user at a higher permission level.Thus, different users will have different capabilities with regard toaccessing and modifying application and database information, dependingon a user's security or permission level.

Network 614 is any network or combination of networks of devices thatcommunicate with one another. For example, network 614 can be any one orany combination of a LAN (local area network), WAN (wide area network),telephone network, wireless network, point-to-point network, starnetwork, token ring network, hub network, or other appropriateconfiguration. As the most common type of computer network in currentuse is a TCP/IP (Transfer Control Protocol and Internet Protocol)network, such as the global internetwork of networks often referred toas the “Internet” with a capital “I,” that network will be used in manyof the examples herein. However, it should be understood that thenetworks that one or more implementations might use are not so limited,although TCP/IP is a frequently implemented protocol.

User systems 612 might communicate with system 616 using TCP/IP and, ata higher network level, use other common Internet protocols tocommunicate, such as HTTP, FTP, AFS, WAP, etc. In an example where HTTPis used, user system 612 might include an HTTP client commonly referredto as a “browser” for sending and receiving HTTP messages to and from anHTTP server at system 616. Such an HTTP server might be implemented asthe sole network interface between system 616 and network 614, but othertechniques might be used as well or instead. In some implementations,the interface between system 616 and network 614 includes load sharingfunctionality, such as round-robin HTTP request distributors to balanceloads and distribute incoming HTTP requests evenly over a plurality ofservers. At least as for the users that are accessing that server, eachof the plurality of servers has access to the MTS' data; however, otheralternative configurations may be used instead.

In one embodiment, system 616, shown in FIG. 6, implements a web-basedcustomer relationship management (CRM) system. For example, in oneembodiment, system 616 includes application servers configured toimplement and execute CRM software applications as well as providerelated data, code, forms, webpages and other information to and fromuser systems 612 and to store to, and retrieve from, a database systemrelated data, objects, and Webpage content. With a multi-tenant system,data for multiple tenants may be stored in the same physical databaseobject, however, tenant data typically is arranged so that data of onetenant is kept logically separate from that of other tenants so that onetenant does not have access to another tenant's data, unless such datais expressly shared. In certain embodiments, system 616 implementsapplications other than, or in addition to, a CRM application. Forexample, system 616 may provide tenant access to multiple hosted(standard and custom) applications, including a CRM application. User(or third party developer) applications, which may or may not includeCRM, may be supported by the application platform 618, which managescreation, storage of the applications into one or more database objectsand executing of the applications in a virtual machine in the processspace of the system 616.

One arrangement for elements of system 616 is shown in FIG. 6, includinga network interface 620, application platform 618, tenant data storage622 for tenant data 623, system data storage 624 for system data 625accessible to system 616 and possibly multiple tenants, program code 626for implementing various functions of system 616, and a process space628 for executing MTS system processes and tenant-specific processes,such as running applications as part of an application hosting service.Additional processes that may execute on system 616 include databaseindexing processes.

Several elements in the system shown in FIG. 6 include conventional,well-known elements that are explained only briefly here. For example,each user system 612 could include a desktop personal computer,workstation, laptop, PDA, cell phone, or any wireless access protocol(WAP) enabled device or any other computing device capable ofinterfacing directly or indirectly to the Internet or other networkconnection. User system 612 typically runs an HTTP client, e.g., abrowsing program, such as Microsoft's Internet Explorer browser,Netscape's Navigator browser, Opera's browser, or a WAP-enabled browserin the case of a cell phone, PDA or other wireless device, or the like,allowing a user (e.g., subscriber of the multi-tenant database system)of user system 612 to access, process and view information, pages andapplications available to it from system 616 over network 614. Usersystem 612 further includes Mobile OS (e.g., iOS® by Apple®, Android®,WebOS® by Palm®, etc.). Each user system 612 also typically includes oneor more user interface devices, such as a keyboard, a mouse, trackball,touch pad, touch screen, pen or the like, for interacting with agraphical user interface (GUI) provided by the browser on a display(e.g., a monitor screen, LCD display, etc.) in conjunction with pages,forms, applications and other information provided by system 616 orother systems or servers. For example, the user interface device can beused to access data and applications hosted by system 616, and toperform searches on stored data, and otherwise allow a user to interactwith various GUI pages that may be presented to a user. As discussedabove, embodiments are suitable for use with the Internet, which refersto a specific global internetwork of networks. However, it should beunderstood that other networks can be used instead of the Internet, suchas an intranet, an extranet, a virtual private network (VPN), anon-TCP/IP based network, any LAN or WAN or the like.

According to one embodiment, each user system 612 and all of itscomponents are operator configurable using applications, such as abrowser, including computer code run using a central processing unitsuch as an Intel Core® processor or the like. Similarly, system 616 (andadditional instances of an MTS, where more than one is present) and allof their components might be operator configurable using application(s)including computer code to run using a central processing unit such asprocessor system 617, which may include an Intel Pentium® processor orthe like, and/or multiple processor units. A computer program productembodiment includes a machine-readable storage medium (media) havinginstructions stored thereon/in which can be used to program a computerto perform any of the processes of the embodiments described herein.Computer code for operating and configuring system 616 tointercommunicate and to process webpages, applications and other dataand media content as described herein are preferably downloaded andstored on a hard disk, but the entire program code, or portions thereof,may also be stored in any other volatile or non-volatile memory mediumor device as is well known, such as a ROM or RAM, or provided on anymedia capable of storing program code, such as any type of rotatingmedia including floppy disks, optical discs, digital versatile disk(DVD), compact disk (CD), microdrive, and magneto-optical disks, andmagnetic or optical cards, nanosystems (including molecular memory ICs),or any type of media or device suitable for storing instructions and/ordata. Additionally, the entire program code, or portions thereof, may betransmitted and downloaded from a software source over a transmissionmedium, e.g., over the Internet, or from another server, as is wellknown, or transmitted over any other conventional network connection asis well known (e.g., extranet, VPN, LAN, etc.) using any communicationmedium and protocols (e.g., TCP/IP, HTTP, HTTPS, Ethernet, etc.) as arewell known. It will also be appreciated that computer code forimplementing embodiments can be implemented in any programming languagethat can be executed on a client system and/or server or server systemsuch as, for example, C, C++, HTML, any other markup language, Java™JavaScript, ActiveX, any other scripting language, such as VBScript, andmany other programming languages as are well known may be used. (Java™is a trademark of Sun Microsystems, Inc.).

According to one embodiment, each system 616 is configured to providewebpages, forms, applications, data and media content to user (client)systems 612 to support the access by user systems 612 as tenants ofsystem 616. As such, system 616 provides security mechanisms to keepeach tenant's data separate unless the data is shared. If more than oneMTS is used, they may be located in close proximity to one another(e.g., in a server farm located in a single building or campus), or theymay be distributed at locations remote from one another (e.g., one ormore servers located in city A and one or more servers located in cityB). As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” is meant toinclude a computer system, including processing hardware and processspace(s), and an associated storage system and database application(e.g., OODBMS or RDBMS) as is well known in the art. It should also beunderstood that “server system” and “server” are often usedinterchangeably herein. Similarly, the database object described hereincan be implemented as single databases, a distributed database, acollection of distributed databases, a database with redundant online oroffline backups or other redundancies, etc., and might include adistributed database or storage network and associated processingintelligence.

FIG. 7 also illustrates environment 610. However, in FIG. 7 elements ofsystem 616 and various interconnections in an embodiment are furtherillustrated. FIG. 7 shows that user system 612 may include processorsystem 612A, memory system 612B, input system 612C, and output system612D. FIG. 7 shows network 614 and system 616. FIG. 7 also shows thatsystem 616 may include tenant data storage 622, tenant data 623, systemdata storage 624, system data 625, User Interface (UI) 730, ApplicationProgram Interface (API) 732, PL/SOQL 734, save routines 736, applicationsetup mechanism 738, applications servers 700 ₁-700 _(N), system processspace 702, tenant process spaces 704, tenant management process space710, tenant storage area 712, user storage 714, and application metadata716. In other embodiments, environment 610 may not have the sameelements as those listed above and/or may have other elements insteadof, or in addition to, those listed above.

User system 612, network 614, system 616, tenant data storage 622, andsystem data storage 624 were discussed above in FIG. 6. Regarding usersystem 612, processor system 612A may be any combination of one or moreprocessors. Memory system 612B may be any combination of one or morememory devices, short term, and/or long term memory. Input system 612Cmay be any combination of input devices, such as one or more keyboards,mice, trackballs, scanners, cameras, and/or interfaces to networks.Output system 612D may be any combination of output devices, such as oneor more monitors, printers, and/or interfaces to networks. As shown byFIG. 7, system 616 may include a network interface 620 (of FIG. 6)implemented as a set of HTTP application servers 700, an applicationplatform 618, tenant data storage 622, and system data storage 624. Alsoshown is system process space 702, including individual tenant processspaces 704 and a tenant management process space 710. Each applicationserver 700 may be configured to tenant data storage 622 and the tenantdata 623 therein, and system data storage 624 and the system data 625therein to serve requests of user systems 612. The tenant data 623 mightbe divided into individual tenant storage areas 712, which can be eithera physical arrangement and/or a logical arrangement of data. Within eachtenant storage area 712, user storage 714 and application metadata 716might be similarly allocated for each user. For example, a copy of auser's most recently used (MRU) items might be stored to user storage714. Similarly, a copy of MRU items for an entire organization that is atenant might be stored to tenant storage area 712. A UI 730 provides auser interface and an API 732 provides an application programmerinterface to system 616 resident processes to users and/or developers atuser systems 612. The tenant data and the system data may be stored invarious databases, such as one or more Oracle™ databases.

Application platform 618 includes an application setup mechanism 738that supports application developers' creation and management ofapplications, which may be saved as metadata into tenant data storage622 by save routines 736 for execution by subscribers as one or moretenant process spaces 704 managed by tenant management process 710 forexample. Invocations to such applications may be coded using PL/SOQL 734that provides a programming language style interface extension to API732. A detailed description of some PL/SOQL language embodiments isdiscussed in commonly owned U.S. Pat. No. 7,730,478 entitled, “Methodand System for Allowing Access to Developed Applicants via aMulti-Tenant Database On-Demand Database Service”, issued Jun. 1, 2010to Craig Weissman, which is incorporated in its entirety herein for allpurposes. Invocations to applications may be detected by one or moresystem processes, which manage retrieving application metadata 716 forthe subscriber making the invocation and executing the metadata as anapplication in a virtual machine.

Each application server 700 may be communicably coupled to databasesystems, e.g., having access to system data 625 and tenant data 623, viaa different network connection. For example, one application server 700₁ might be coupled via the network 614 (e.g., the Internet), anotherapplication server 700 _(N-1) might be coupled via a direct networklink, and another application server 700 _(N) might be coupled by yet adifferent network connection. Transfer Control Protocol and InternetProtocol (TCP/IP) are typical protocols for communicating betweenapplication servers 700 and the database system. However, it will beapparent to one skilled in the art that other transport protocols may beused to optimize the system depending on the network interconnect used.

In certain embodiments, each application server 700 is configured tohandle requests for any user associated with any organization that is atenant. Because it is desirable to be able to add and remove applicationservers from the server pool at any time for any reason, there ispreferably no server affinity for a user and/or organization to aspecific application server 700. In one embodiment, therefore, aninterface system implementing a load balancing function (e.g., an F5Big-IP load balancer) is communicably coupled between the applicationservers 700 and the user systems 612 to distribute requests to theapplication servers 700. In one embodiment, the load balancer uses aleast connections algorithm to route user requests to the applicationservers 700. Other examples of load balancing algorithms, such as roundrobin and observed response time, also can be used. For example, incertain embodiments, three consecutive requests from the same user couldhit three different application servers 700, and three requests fromdifferent users could hit the same application server 700. In thismanner, system 616 is multi-tenant, wherein system 616 handles storageof, and access to, different objects, data and applications acrossdisparate users and organizations.

As an example of storage, one tenant might be a company that employs asales force where each salesperson uses system 616 to manage their salesprocess. Thus, a user might maintain contact data, leads data, customerfollow-up data, performance data, goals and progress data, etc., allapplicable to that user's personal sales process (e.g., in tenant datastorage 622). In an example of a MTS arrangement, since all of the dataand the applications to access, view, modify, report, transmit,calculate, etc., can be maintained and accessed by a user system havingnothing more than network access, the user can manage his or her salesefforts and cycles from any of many different user systems. For example,if a salesperson is visiting a customer and the customer has Internetaccess in their lobby, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by system 616 that are allocatedat the tenant level while other data structures might be managed at theuser level. Because an MTS might support multiple tenants includingpossible competitors, the MTS should have security protocols that keepdata, applications, and application use separate. Also, because manytenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in the MTS. In addition to user-specific data andtenant specific data, system 616 might also maintain system level datausable by multiple tenants or other data. Such system level data mightinclude industry reports, news, postings, and the like that are sharableamong tenants.

In certain embodiments, user systems 612 (which may be client systems)communicate with application servers 700 to request and updatesystem-level and tenant-level data from system 616 that may requiresending one or more queries to tenant data storage 622 and/or systemdata storage 624. System 616 (e.g., an application server 700 in system616) automatically generates one or more SQL statements (e.g., one ormore SQL queries) that are designed to access the desired information.System data storage 624 may generate query plans to access the requesteddata from the database.

Each database can generally be viewed as a collection of objects, suchas a set of logical tables, containing data fitted into predefinedcategories. A “table” is one representation of a data object, and may beused herein to simplify the conceptual description of objects and customobjects. It should be understood that “table” and “object” may be usedinterchangeably herein. Each table generally contains one or more datacategories logically arranged as columns or fields in a viewable schema.Each row or record of a table contains an instance of data for eachcategory defined by the fields. For example, a CRM database may includea table that describes a customer with fields for basic contactinformation such as name, address, phone number, fax number, etc.Another table might describe a purchase order, including fields forinformation such as customer, product, sale price, date, etc. In somemulti-tenant database systems, standard entity tables might be providedfor use by all tenants. For CRM database applications, such standardentities might include tables for Account, Contact, Lead, andOpportunity data, each containing pre-defined fields. It should beunderstood that the word “entity” may also be used interchangeablyherein with “object” and “table”.

In some multi-tenant database systems, tenants may be allowed to createand store custom objects, or they may be allowed to customize standardentities or objects, for example by creating custom fields for standardobjects, including custom index fields. U.S. patent application Ser. No.10/817,161, filed Apr. 2, 2004, entitled “Custom Entities and Fields ina Multi-Tenant Database System”, and which is hereby incorporated hereinby reference, teaches systems and methods for creating custom objects aswell as customizing standard objects in a multi-tenant database system.In certain embodiments, for example, all custom entity data rows arestored in a single multi-tenant physical table, which may containmultiple logical tables per organization. It is transparent to customersthat their multiple “tables” are in fact stored in one large table orthat their data may be stored in the same table as the data of othercustomers.

Any of the above embodiments may be used alone or together with oneanother in any combination. Embodiments encompassed within thisspecification may also include embodiments that are only partiallymentioned or alluded to or are not mentioned or alluded to at all inthis brief summary or in the abstract. Although various embodiments mayhave been motivated by various deficiencies with the prior art, whichmay be discussed or alluded to in one or more places in thespecification, the embodiments do not necessarily address any of thesedeficiencies. In other words, different embodiments may addressdifferent deficiencies that may be discussed in the specification. Someembodiments may only partially address some deficiencies or just onedeficiency that may be discussed in the specification, and someembodiments may not address any of these deficiencies.

While one or more implementations have been described by way of exampleand in terms of the specific embodiments, it is to be understood thatone or more implementations are not limited to the disclosedembodiments. To the contrary, it is intended to cover variousmodifications and similar arrangements as would be apparent to thoseskilled in the art. Therefore, the scope of the appended claims shouldbe accorded the broadest interpretation so as to encompass all suchmodifications and similar arrangements. It is to be understood that theabove description is intended to be illustrative, and not restrictive.

What is claimed is:
 1. A method comprising: fetching, by a queueresource-management server computing device of a database system, tenanthierarchies associated with tenants and message type hierarchiesassociated with message types in a multi-level hierarchical formation,wherein the tenant hierarchies and the message type hierarchies arebased on multiple dimensions associated with the tenants and the messagetypes, respectively, wherein the multiple dimensions are considered in ahierarchical order based on the multi-level hierarchical formation andinclude current queuing time associated with the tenants and the messagetypes and one or more of size and service level associated with thetenants and importance and organizational function associated with themessage types; receiving, in runtime, snapshots of queues associatedwith the tenants and the message types to indicate queuing timesassociated with the tenants and the message types; assigning, inruntime, weights to at least one of the tenants and the message typesbased on the queuing times associated with the tenants and the messagetypes and further based on the tenant hierarchies and the message typehierarchies, respectively, wherein the assigned weights are dynamicallyscaled, in runtime, based on changes to one or more of the multipledimensions associated with at least one of the tenants and the messagetypes; and allocating, in runtime, resources to one or more of thetenants and one or more of the message types based on the multi-levelhierarchical formation and one or more of the weights assigned to theone or more tenants and the one or more message types, wherein the oneor more of the tenants or the one or more of the message types arepromoted or demoted between the queues such that the allocated resourcesare dynamically balanced or modified, in runtime, based on scaling ofthe one or more weights and changes to the multi-level hierarchicalformation.
 2. The method of claim 1, wherein fetching further comprisesretrieving queuing time associated with at least one of each of themessage types and each of the tenants.
 3. The method of claim 1, furthercomprising allocating zero resources to at least one of one or more zeroresource-consuming tenants of the tenants and one or more zeroresource-consuming message types of the message types that aredetermined as using zero amount of the queuing time.
 4. The method ofclaim 1, wherein fetching further comprises retrieving the weights froma weights cache, wherein the weights are further to identify one or moreof the multiple dimensions associated with the tenants or the messagetypes, wherein the multiple dimensions further include one or more ofpriority levels and process urgency levels associated with the tenantsand the message types.
 5. The method of claim 1, further comprising:monitoring resource consumption levels associated with the tenants andthe message types, wherein the resource consumption levels are monitoredto ensure fair usage of the resources by at least one of the tenants andthe message types based on the assigned weights; and computing theweights prior to assigning the weights to at least one of the tenantsand the message types.
 6. A system comprising: a database system havinga queue resource-management server computing device having a processingdevice coupled to memory, the processing device to facilitate amechanism to perform operations comprising: fetching tenant hierarchiesassociated with tenants and message type hierarchies associated withmessage types in a multi-level hierarchical formation, wherein thetenant hierarchies and the message type hierarchies are based onmultiple dimensions associated with the tenants and the message types,respectively, wherein the multiple dimensions are considered in ahierarchical order based on the multi-level hierarchical formation andinclude current queuing time associated with the tenants and the messagetypes and one or more of size and service level associated with thetenants and importance and organizational function associated with themessage types; receiving, in runtime, snapshots of queues associatedwith the tenants and the message types to indicate queuing timesassociated with the tenants and the message types; assigning, inruntime, weights to at least one of the tenants and the message typesbased on the queuing times associated with the tenants and the messagetypes and further based on the tenant hierarchies and the message typehierarchies, respectively, wherein the assigned weights are dynamicallyscaled, in runtime, based on changes to one or more of the multipledimensions associated with at least one of the tenants and the messagetypes; and allocating, in runtime, resources to one or more of thetenants and one or more of the message types based on the multi-levelhierarchical formation and one or more of the weights assigned to theone or more tenants and the one or more message types, wherein the oneor more of the tenants or the one or more of the message types arepromoted or demoted between the queues such that the allocated resourcesare dynamically balanced or modified, in runtime, based on scaling ofthe one or more weights and changes to the multi-level hierarchicalformation.
 7. The system of claim 6, wherein fetching further comprisesretrieving queuing time associated with at least one of each of themessage types and each of the tenants.
 8. The system of claim 6, whereinthe operations further comprise allocating zero resources to at leastone of one or more zero resource-consuming tenants of the tenants andone or more zero resource-consuming message types of the message typesthat are determined as using zero amount of the queuing time.
 9. Thesystem of claim 6, wherein fetching further comprises retrieving theweights from a weights cache, wherein the weights are further toidentify one or more of the multiple dimensions associated with thetenants or the message types, wherein the multiple dimensions furtherinclude one or more of priority levels and process urgency levelsassociated with the tenants and the message types.
 10. The system ofclaim 6, wherein the operations further comprise: monitoring resourceconsumption levels associated with the tenants and the message types,wherein the resource consumption levels are monitored to ensure fairusage of the resources by at least one of the tenants and the messagetypes based on the assigned weights; and computing the weights prior toassigning the weights to at least one of the tenants and the messagetypes.
 11. A non-transitory machine-readable medium having storedthereon instructions which, when executed by a processing device, causethe processing device to perform operations comprising: fetching tenanthierarchies associated with tenants and message type hierarchiesassociated with message types in a multi-level hierarchical formation,wherein the tenant hierarchies and the message type hierarchies arebased on multiple dimensions associated with the tenants and the messagetypes, respectively, wherein the multiple dimensions are considered in ahierarchical order based on the multi-level hierarchical formation andinclude current queuing time associated with the tenants and the messagetypes and one or more of size and service level associated with thetenants and importance and organizational function associated with themessage types; receiving, in runtime, snapshots of queues associatedwith the tenants and the message types to indicate queuing timesassociated with the tenants and the message types; assigning, inruntime, weights to at least one of the tenants and the message typesbased on the queuing times associated with the tenants and the messagetypes and further based on the tenant hierarchies and the message typehierarchies, respectively, wherein the assigned weights are dynamicallyscaled, in runtime, based on changes to one or more of the multipledimensions associated with at least one of the tenants and the messagetypes; and allocating, in runtime, resources to one or more of thetenants and one or more of the message types based on the multi-levelhierarchical formation and one or more of the weights assigned to theone or more tenants and the one or more message types, wherein the oneor more of the tenants or the one or more of the message types arepromoted or demoted between the queues such that the allocated resourcesare dynamically balanced or modified, in runtime, based on scaling ofthe one or more weights and changes to the multi-level hierarchicalformation.
 12. The non-transitory machine-readable medium of claim 11,wherein fetching further comprises retrieving queuing time associatedwith at least one of each of the message types and each of the tenants.13. The non-transitory machine-readable medium of claim 11, wherein theoperations further comprise allocating zero resources to at least one ofone or more zero resource-consuming tenants of the tenants and one ormore zero resource-consuming message types of the message types that aredetermined as using zero amount of the queuing time.
 14. Thenon-transitory machine-readable medium of claim 11, wherein fetchingfurther comprises retrieving the weights from a weights cache, whereinthe weights are further to identify one or more of the multipledimensions associated with the tenants or the message types, wherein themultiple dimensions further include one or more of priority levels andprocess urgency levels associated with the tenants and the messagetypes.
 15. The non-transitory machine-readable medium of claim 11,wherein the operations further comprise: monitoring resource consumptionlevels associated with the tenants and the message types, wherein theresource consumption levels are monitored to ensure fair usage of theresources by at least one of the tenants and the message types based onthe assigned weights; and computing the weights prior to assigning theweights to at least one of the tenants and the message types.